The following is a Bar chart illustration of the data in the above table.
This report gives the result of running the computer algebra independent integration test.The download section in on the main webpage contains links to download the problems in plain text format used for all CAS systems. The number of integrals in this report is [ 96 ]. This is test number [ 47 ].
The following are the CAS systems tested:
Maxima and Fricas and Giac are called using Sagemath. This was done using Sagemath integrate command by changing the name of the algorithm to use the different CAS systems.
Sympy was run directly in Python not via sagemath.
Important note: A number of problems in this test suite have no antiderivative in closed form. This means the antiderivative of these integrals can not be expressed in terms of elementary, special functions or Hypergeometric2F1 functions. RootSum and RootOf are not allowed. If a CAS returns the above integral unevaluated within the time limit, then the result is counted as passed and assigned an A grade.
However, if CAS times out, then it is assigned an F grade even if the integral is not integrable, as this implies CAS could not determine that the integral is not integrable in the time limit.
If a CAS returns an antiderivative to such an integral, it is assigned an A grade automatically and this special result is listed in the introduction section of each individual test report to make it easy to identify as this can be important result to investigate.
The results given in in the table below reflects the above.
| System | % solved | % Failed |
| Rubi | 100.00 ( 96 ) | 0.00 ( 0 ) |
| Mathematica | 95.83 ( 92 ) | 4.17 ( 4 ) |
| Maple | 51.04 ( 49 ) | 48.96 ( 47 ) |
| Fricas | 51.04 ( 49 ) | 48.96 ( 47 ) |
| Mupad | 51.04 ( 49 ) | 48.96 ( 47 ) |
| Sympy | 43.75 ( 42 ) | 56.25 ( 54 ) |
| Giac | 38.54 ( 37 ) | 61.46 ( 59 ) |
| Maxima | 17.71 ( 17 ) | 82.29 ( 79 ) |
The table below gives additional break down of the grading of quality of the antiderivatives generated by each CAS. The grading is given using the letters A,B,C and F with A being the best quality. The grading is accomplished by comparing the antiderivative generated with the optimal antiderivatives included in the test suite. The following table describes the meaning of these grades.
| grade |
description |
| A |
Integral was solved and antiderivative is optimal in quality and leaf size. |
| B |
Integral was solved and antiderivative is optimal in quality but leaf size is larger than twice the optimal antiderivatives leaf size. |
| C |
Integral was solved and antiderivative is non-optimal in quality. This can be due to one or more of the following reasons
|
| F |
Integral was not solved. Either the integral was returned unevaluated within the time limit, or it timed out, or CAS hanged or crashed or an exception was raised. |
Grading is implemented for all CAS systems. Based on the above, the following table summarizes the grading for this test suite.
| System | % A grade | % B grade | % C grade | % F grade |
| Rubi | 94.792 | 0.000 | 0.000 | 5.208 |
| Mathematica | 57.292 | 11.458 | 21.875 | 9.375 |
| Sympy | 29.167 | 6.250 | 8.333 | 56.250 |
| Giac | 27.083 | 6.250 | 0.000 | 66.667 |
| Maxima | 12.500 | 0.000 | 0.000 | 87.500 |
| Fricas | 10.417 | 29.167 | 6.250 | 54.167 |
| Maple | 7.292 | 0.000 | 38.542 | 54.167 |
| Mupad | 0.000 | 45.833 | 0.000 | 54.167 |
The following is a Bar chart illustration of the data in the above table.
The figure below compares the grades of the CAS systems.
The following table shows the distribution of the different types of failures for each CAS. There are 3 types failures. The first is when CAS returns the input within the time limit, which means it could not solve it. This is the typical failure and given as F.
The second failure is due to time out. CAS could not solve the integral within the 3 minutes time limit which is assigned. This is assigned F(-1).
The third is due to an exception generated, indicated as F(-2). This most likely indicates an interface problem between sagemath and the CAS (applicable only to FriCAS, Maxima and Giac) or it could be an indication of an internal error in the CAS itself. This type of error requires more investigation to determine the cause.
| System |
Number failed |
Percentage normal failure |
Percentage timeout failure |
Percentage exception failure |
| Rubi |
0 |
0.00 |
0.00 |
0.00 |
| Mathematica |
4 |
100.00 |
0.00 |
0.00 |
| Fricas |
47 | 89.36 | 0.00 | 10.64 |
| Maple | 47 | 100.00 | 0.00 | 0.00 |
| Mupad |
47 |
0.00 |
100.00 |
0.00 |
| Sympy |
54 |
16.67 |
68.52 |
14.81 |
| Giac |
59 |
91.53 |
1.69 |
6.78 |
| Maxima |
79 |
98.73 |
0.00 |
1.27 |
The table below summarizes the performance of each CAS system in terms of time used and leaf size of results.
Mean size is the average leaf size produced by the CAS (before any normalization). The Normalized mean is relative to the mean size of the optimal anti-derivative given in the input files.
For example, if CAS has Normalized mean of \(3\), then the mean size of its leaf size is 3 times as large as the mean size of the optimal leaf size.
Median size is value of leaf size where half the values are larger than this and half are smaller (before any normalization). i.e. The Middle value.
Similarly the Normalized median is relative to the median leaf size of the optimal.
For example, if a CAS has Normalized median of \(1.2\), then its median is \(1.2\) as large as the median leaf size of the optimal.
| System |
Mean size |
Normalized mean |
Median size |
Normalized median |
| Maple |
51.29 |
0.41 |
44.00 |
0.26 |
| Maxima |
128.29 |
1.10 |
72.00 |
1.08 |
| Giac |
331.35 | 1.75 | 147.00 | 0.94 |
| Rubi | 363.12 | 1.00 | 288.00 | 1.00 |
| Sympy |
435.00 |
2.78 |
110.50 |
0.48 |
| Fricas |
1192.80 |
3.44 |
349.00 |
2.21 |
| Mathematica |
2014.07 |
2.04 |
134.50 |
0.90 |
| Mupad |
2816.22 |
6.61 |
269.00 |
1.48 |
This section shows how each CAS performed based on the number of rules Rubi needed to solve the same integral. One diagram is given for each CAS.
On the \(y\) axis is the percentage solved which Rubi itself needed the number of rules given the \(x\) axis. These plots show that as more rules are needed then most CAS system percentage of solving decreases which indicates the integral is becoming more complicated to solve.
This section shows how each CAS performed based on the number of steps Rubi needed to solve the same integral. Note that the number of steps Rubi needed can be much higher than the number of rules, as the same rule could be used more than once.
The above diagram show that the precentage of solved intergals decreases for most CAS systems as the number of steps increases. As expected, for integrals that required less steps by Rubi, CAS systems had more success which indicates the integral was not as hard to solve. As Rubi needed more steps to solve the integral, the solved percentage decreased for most CAS systems which indicates the integral is becoming harder to solve.
The following shows the distribution of solved integrals for each CAS system based on leaf size of the antiderivatives produced by each CAS. It shows that most integrals solved produced leaf size less than about 100 to 150. The bin size used is \(40\).
The following shows the distribution of solved integrals for each CAS system based on CPU time used in seconds. The bin size used is \(0.1\) second.
The following shows the relation between the CPU time used to solve an integral and the leaf size of the antiderivative.
The result for Fricas, Maxima and Giac is shifted more to the right than the other CAS system due to the use of sagemath to call them, which causes an initial slight delay in the timing to start the integration due to overhead of starting a new process each time. This should also be taken into account when looking at the timing of these three CAS systems. Direct calls not using sagemath would result in faster timings, but current implementation uses sagemath as this makes testing much easier to do.
Rubi
{}
Mathematica
{}
Maple
{}
Maxima
{}
Fricas
{}
Sympy
{}
Giac
{}
Mupad
{}
The following are integrals solved by CAS but the verification phase failed to verify the anti-derivative produced is correct. This does not necessarily mean that the anti-derivative is wrong as additional methods of verification might be needed, or more time is needed (3 minutes time limit was used). These integrals are listed here to make it possible to do further investigation to determine why the result could not be verified.
Rubi
{}
Mathematica
Maple
{}
Maxima
Verification phase not currently implemented.
Fricas
Verification phase not currently implemented.
Sympy
Verification phase not currently implemented.
Giac
Verification phase not currently implemented.
Mupad
Verification phase not currently implemented.
The command AbsoluteTiming[] was used in Mathematica to obtain the elapsed time for each integrate call. In Maple, the command Usage was used as in the following example
cpu_time := Usage(assign ('result_of_int',int(expr,x)),output='realtime'
For all other CAS systems, the elapsed time to complete each integral was found by taking the difference between the time after the call completed from the time before the call was made. This was done using Python’s time.time() call.
All elapsed times shown are in seconds. A time limit of 3 CPU minutes was used for each integral. If the integrate command did not complete within this time limit, the integral was aborted and considered to have failed and assigned an F grade. The time used by failed integrals due to time out was not counted in the final statistics.
A verification phase was applied on the result of integration for Rubi and Mathematica.
Future version of this report will implement verification for the other CAS systems. For the integrals whose result was not run through a verification phase, it is assumed that the antiderivative was correct.
Verification phase also had 3 minutes time out. An integral whose result was not verified could still be correct, but further investigation is needed on those integrals. These integrals were marked in the summary table below and also in each integral separate section so they are easy to identify and locate.
The following diagram gives a high level view of the current test build system.
